Convergent Reinforcement Learning for Hierarchical Reactive Plans

Hierarchical reinforcement learning techniques operate on structured plans. Although structured representations add expressive power to Markov Decision Processes (MDPs), current approaches impose constraints that force the associated convergence proofs to depend upon a subroutinestyle execution model that restricts adaptive response. We develop an alternate approach to convergent learning that employs hierarchical plans with an interruptible (reactive) execution model. We prove this format reduces to an MDP, and thus that we can implement any algorithm that converges on MDPs directly on reactive plans. We introduce one such algorithm, called Sharsha(0) (an acronym for state hierarchy, action, state hierarchy, action) that simultaneously finds the Q-values for subplans and the optimal hierarchical policy via an on-line exploration of a single, infinitely long execution sequence.